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Materiel  Solutions  Should  Co-Evolve  Simultaneously 
with  User’s  Concept  of  Operation 


Users  at  All  Echelons 


_ y 

Soldier-centric  battlefield  performance  at 
operational,  strategic,  and  tactical  levels. 


Finding  the  sweet-spot  among  competing 
objectives  (performance,  unit  cost,  O&S  costs, 
development  risk,  and  growth  potential)  is  a  non¬ 
trivial  task. 


•  No  Existing  Way  to  Measure  Battlefield  Impact  of  Tradespace  Choices 

•  We  have  only  stochastic  (non  Soldier-in-the-loop)  sims 
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..But  they  PONT 


How  do  you  develop  a  system  if  you  do  not  know  what  it  is  supposed  to  do? 


CONOPS  =  Concept  of  Operation 


108  SE's  surveyed  (18  POD  Orqs.  and  Major  Contractors) 

•  36%  never  worked  a  program  with  a  CONOP 

•  73%  did  not  complete  CONOPS  by  program  start 

•  50%  did  not  update  CONOPS 

•  30%  did  not  even  involve  a  user 


60  CONOPS  examined: 

•  took  3-30  months  to  complete 

•  25%  did  not  state  mission  needs 

•  80%  did  not  discuss  system  risks 

•  50%  did  not  include  operational  scenarios 


SOURCE:  Roberts,  N.,  &  Edson,  R.  (2008).  System  Concept  of  Operations:  Standards ,  Practices  and 
Reality.  NDIA  Systems  Engineering  Conference,  San  Diego  CA. 
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One  Solution:  Use  a  Physics-Based  Game  Environment 


Tardec 
Virtual 
Experiments 
Capability 

■  VBS3  Training  Game 

■  <75  Soldier  Experiments 

■  2-3  Days  =  Several  Refights 

■  Lickert  Subjective  Questionaires 


Virtual  Battlespace  3  (VBS3) 

•  Drag-and-drop  training  game 

•  Hyper-realistic  (i.e.  physics) 

•  https://bisimulations.  com / 


Army  Games  for  Training:  http://www.peostri.army.mil/PRODUCTS/USAGFTP/ 
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TARDEC’s  Digital -Prototyping  Process 


Digital  Prototyping  and 
Experimentation 


OBJECTIVE  DATA 

-and- 

SUBJECTIVE  SURVEYS 


Engineering  Concepts 
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TARDEC’s  Digital -Prototyping  Process 


COMBAT  EXPERIENCE  IN  FUTURE 


HIGHER  FIDELITY  AS 
NEEDED  (SMALLER  DATASET) 


OBJECTIVE  DATA 

-and- 

SUBJECTIVE  SURVEYS 


Digital  Prototyping  and 
Experimentation 


Engineering  Concepts 
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TVEC  NEXT  GENERATION  CLOSE  COMBAT  VEHICLE 
STUDY  VIRTUAL  DEMONSTRATOR  TEST. 

Ft.  Bliss,  Brigade  Modernization  Command,  Dec.  2014 


TEST  DESIGN 

•  What  would  motivate  Soldiers  to  participate? 

•  How  to  get  data  useful  to  concept  developers? 

•  76  Soldiers  over  two  days 

•  Soldier  vs.  Soldier 

AIRFIELD  SEIZURE  MISSION 

•  Airborne  unit  jumps  in  with  NGCCVs 

•  Goal  capture  the  airfield  to  land  heavier  assets 

•  Framed  in  a  way  that  made  it  feel  more  like  a  game  to 
participants 


Remote  Weapon 
Station 


External  Fully  Active  Suspension 


Advanced 

Combat 

Engine 


TEST  EXECUTION 

•  4  rows  of  6  workstations 

•  BLUFOR  on  left,  OPFOR  on  right 

•  Used  either  a  mouse  and  keyboard  or 
steering  wheel  and  pedals,  if  driving 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 
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TVEC  SQUAD  CENTRIC  MOUNTED  MANUEVER  (SCMM) 
VIRTUAL  DEMONSTRATOR  TEST 

Ft.  Hood  1st  Cav,  June  2015 


Ft.  Hood  1st  Cav,  June  2015 

•  Hands-on  experience  operating  “Fire  Team”  vehicles 

•  Iterate  between  identical  physical  and  virtual 

•  Explore  how  Soldiers  might  operate 

•  Soldier  vs.  Soldier 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 
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[ESP  Systems  Engineering  Construct 
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Mission  Set 


High-Fidelity  CAE  Models 
And  Concepts  (PHYSICS) 
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EARLY  SYNTHETIC  PROTOTYPING 
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Robust  /  Adaptable  System 
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Example  Tactical  Utility  Analysis  of  Alternatives 

(Based  on  1400  MindRover  Runs  by  Cadets) 

Highest  tactical  performance  per 
cost/mass/schedule/etc 

200%  ““ 

150% 

EFFECTIVENESS 
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Each  Variant  was  Fought  Very 
Differently 


Source:  "Demonstration  of  Decision  Support  Tools  for  Evaluating  Ground  Combat  System  Survivability,  Lethality,  and 

Mobility  at  the  Tactical,  Operational,  and  Strategic  Levels  of  War."  MAJ  Joshua  Keena  PhD  Dissertation.  University  of 
Texas  as  Austin,  August  2011.  https://repositories.lib.utexas.edu/handle/2152/ETD-UT-2011-08-3781 
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Deeper  Understanding  Requires  Game  Analytics 


Virtual  Physics-Based 
Gaming  Environment 


~12  million  hours  of  Soldier 
gameplay  per  year 


Data  Mining 

-  What  are  they  doing? 

-  Where  are  they  doing  it? 

-  Why  they  are  doing  it? 

-  How  effective  is  this? 

-  Terrain  versus  movement  choices 

-  What  are  they  talking  about/  when/  how  often 

-  Optimal  Force  structure 


Engagement  Sector  Cardioid 


Decision  Trees 


Temporal  Usag 


Retrospective 


Heat  Maps 


Multi-Run  Movement  Plots 


Game  Image  Source:  Game  Metrics  and  Biometrics:  The  Future  of  Player  Experience  Research. 
http://www.slideshare.net/acagamic/game-metrics-and-biometrics-the-future-of-player-experience-research 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 
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Aspects  of  Data  Mining  /  Visualization 


Filtering  the  Data 

•  Tracking  player  experience  levels  and  play  styles  (preferences) 

•  “Seriousness”  detection  versus  screwing  around 

•  Learning  curves  on  technologies 

Labeling  the  Data 

•  Annotating  the  gameplay  (painful) 

Analyzing  the  Data 

•  Level  0:  Directly  visualizing  data  (human  finds  the  “so-what”) 

•  Level  1:  Machine  learning  for  individual  group  goals  /  tactics 

(example:  inverse  reinforcement  learning) 

•  Level  2:  Machine  learning  for  group  goals  /  tactics 

Next  Slides:  SBIR  Research  (Small  Business  Innovative  Research  Grants) 

•  Decisive  Analytics 

•  SoarTech  /  USC  Institute  for  Creative  Studies/  Northeastern  University 

•  Creative  Technologies  Inc.  /  UtopiaCompression  Inc. 

NOTE:  SBIR  TOPIC  is  A15-086  "Tactical  Behavior  Mining  of  a  Soldier-Based  Gaming  Environment" 
https://sbirsource.com/sbir/topics/91575 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 
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SBIR  Examples: 

Surrogate  Dataset:  DOTA-2  Commercial  Game 


•  DOTA-2 

-  Objective 

•2  teams  (Dire  and  Radiant) 

•  5  players  each 

•  Each  team  defends  an  “Ancient” 
building 

•  3  main  “Lanes”  between  strongholds 

-  Game  player  description 

•  Players  are  called  Heroes 

•  111  different  Heroes  available 

•  Each  Hero  has  different 

-  Items  (-equipment) 

-  Spells  (-skills  &  capabilities) 

-  Gold 

•  Players  typically  assigned  a  specific 
role  within  the  team 

-  Similar  to  the  different  roles  soldiers  have 
within  a  unit 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 
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Example  Spatio-Temporal  Machine  Learning 
(Decisive  Analytics) 


Actions  are  inferred  from 
trajectory/state  data 

•  Standing 

•  Walking 

•  Walking  with  weapon  drawn 


Tracks  are  sequences  of 
Actions 


Four  tracks  in  ground  plane 

;2 


Inferred  action  chain  per  track 
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Decisive 

Analytics 


Behaviors  are  clusters  of  similar 
Action  chains 


SOURCE:  Decisive  Analytics  Corporation  Phase  I  SBIR.  Contract 
W56HZV-15-C-0190.  htto://www.dac.us/ 
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Behavior  Spatio-Temporal  Clustering 
(Decisive  Analytics) 


SOURCE:  Decisive  Analytics  Corporation  Phase  I  SBIR.  Contract 
W56HZV-15-C-0190.  htto://www.dac.us/ 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 
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Learned  Behaviors 
(Decisive  Analytics) 
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Actions  in  behavior  21:  5  12  14  16  25 
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Action  Transitio 
Probabilities 


1 


Decisive 

Analytics 


•  Each  chart  represents  a  Behavior 
-  Sequences  of  Actions 
Derived  from  Actions,  tracks  and  state  data 


SOURCE:  Decisive  Analytics  Corporation  Phase  I  SBIR.  Contract 
W56HZV-15-C-0190.  htto://www.dac.us/ 
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Learned  Behaviors 
(Decisive  Analytics) 


Action  Transitio 
Probabilities 


1 


Decisive 

Analytics 


•  Each  chart  represents  a  Behavior 
-  Sequences  of  Actions 
Derived  from  Actions,  tracks  and  state  data 


SOURCE:  Decisive  Analytics  Corporation  Phase  I  SBIR.  Contract 
W56HZV-15-C-0190.  htto://www.dac.us/ 
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Slice/Dice  the  Data 
(Decisive  Analytics) 
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SOURCE:  Decisive  Analytics  Corporation  Phase  I  SBIR.  Contract 
W56HZV-15-C-0190.  htto://www.dac.us/ 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 


18 


Gplayer/  Glyph  Tools  from  Northeastern  Univ. 
(SoarTech) 

G-Player  Cluster  Spatial  Visualization 


Lower  Left  Blue  Group 


Upper  Left  Magenta  Group 


Center  Right  Red  Group 


Glyph  Cluster  State  Visualization 


Center  Right  Red  Group  Lower  Left  Blue  Group  Upper  Left  Magenta  Group 


O 

SOARTECH 

Modeling  human  reasoning. 
Enhancing  human  performance. 


Source:  SoarTech/  ICT /  Northeastern  University  Phase  I  SBIR.  Soldier  Sourcing 

Insights  Early  with  a  Game  Environment  (SSIEGE).  W56HZV-15-C-0188 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 
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State  transition  graph  +  Cluster  of  Sequences  Graph  H  V 
(SoarTech) 
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Glyph  visualization  for  8  entities  in  an  ESP  session 


Glyph:  Visual  Analytics 


State  graph  -  shows  transition  between  states  in  the  game 


Cluster  of  Sequences  -  shows  how  patterns  cluster  in  space  where  distance 
is  how  similar  they  are  (the  more  similar  the  closer) 


O 

SOARTECH 


Modeling  human  reasoning. 
Enhancing  human  performance. 


Source:  SoarTech/  ICT /  Northeastern  University  Phase  I  SBIR.  Soldier  Sourcing 

Insights  Early  with  a  Game  Environment  (SSIEGE).  W56HZV-15-C-0188 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 
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Spatio-Temporal  Machine  Learning 
(CTI/  UtopiaCompression) 


12 


Semantic  context  tells  us  what  a 
player  did  within  a  segment. 

Two  approaches  to  obtain  semantic 
context  for  data: 

*  Supervised  labeling:  show 
SMEs/crowd  images,  video  and 
ask  them  to  label  player  actions; 

•  Unsupervised  labeling:  Ask 
SMEs  to  list  behavior  atoms  of 
interest  and  corresponding  low- 
level  features  and  (try  to  label 
context  autonomously. 
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Source:  Creative  Technologies  Inc..  W56HZV-15-C-0189.  Tactical  Behavior 
Mining  of  a  Soldier-Based  Gaming  Environment. 
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Spatio-Temporal  Machine  Learning 
(CTI/  UtopiaCompression) 


Source:  Creative  Technologies  Inc..  W56HZV-15-C-0189.  Tactical  Behavior 
Mining  of  a  Soldier-Based  Gaming  Environment. 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 
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Positional  Analysis  Over  Time 
(CTI/  UtopiaCompression) 


Archetypes;  (top,  Archetype  5)  Radiant  player  who’s  strategy  is  early: safe, 
mid: hard  and  late: Dire.  Switching  lanes  is  an  unusual  behavior  as  it  leads  to 
lane  imbalance  making  a  win  harder. 

(bottom,  Archetype  2)  Dire  player  who  is  unusually  aggressive,  spending  most 
of  the  game  in  the  Radiant  half. 


UTDPI  AGO  MPRESStQIM 
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Source:  Creative  Technologies  Inc..  W56HZV-15-C-0189.  Tactical  Behavior 
Mining  of  a  Soldier-Based  Gaming  Environment. 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 
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ICT  Demo  ESP  Environment  -  Emotion  Tracking 


•  Players  on  laptops  w/  webcam 

•  Intel's  Perceptual  Computing  SDK  captures  data 
about  the  user's  emotional  state 

•  Seven  emotions  (anger,  contempt,  disgust,  fear,  joy, 
sadness,  and  surprise)  and  three  sentiments 
(positive,  neutral,  and  negative). 

•  Also  record  voice  annotation  of  events 


USC  Institute  for 
Creative  Technologies 


More  info:  http://ict.usc.edu/prototvpes/earlv-svnthetic-prototyping-esp/ 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 
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ICT  Demo  ESP  Environment  -  Other  Prototypes 


Let  Soldiers  make  cost-constrained 
tradespace  optimizations  over  a 
mission  set 


Immersive  reality  impact 

•  How  does  the  physiology  of  the  eye  play  into 
detections? 

•  How  does  play  change? 


USC  Institute  for 
Creative  Technologies 


More  info:  http://ict.usc.edu/prototvpes/earlv-svnthetic-prototyping-esp/ 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 
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Labeling  Data:  Retrospective  Interviewing 


Visualizations 


Re-interpret  behavioral  data  based  on  interview  data 


Fig.  1.  Data-driven  retrospective  interviewing  method  overview. 

Seif  El-Nasr,  M.,  Durga,  S.,  Shiyko,  M.,  and  Sceppa,  C.  (2015).  Data-Driven  Retrospective  Interviewing  (DDRI):  A  Proposed  Methodology  for  Formative  Evaluation  of  Pervasive  Games.  Elsevier 
Entertainment  Computing  Journal.  Impact  Factor  1.65 _ 

How  I  See  this  Working  w  TVEC  (Autonomous  Highlight  Reel): 


Player  i  Spawns  Ptew  1  Picks  up  mines  Pfayer  f  Lays  down  mine  Player  1  watches  pteyer  l  tfftnp  on  the  mine 


System  and  method  for  automated  creation  of  video  game  highlights.  Sony  Entertainment.  D  Cottrell  -  US  Patent  8,515,253,  2013. 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited.  26 


Autonomous  Generation  of  Mission  Graphics 


•  Closely  related,  it  would  be  useful  to  communicate  the  battle 
overview  via  military  graphics  discovered  by  data  mining  (combined 
with  retrospective  interviewing). 


Figure  B-6.  Follow  and  assume  tactical  mission  graphic 


Figure  4-10.  Search  and  attack  technique. 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 
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A  1st  Armored  Division  "Old 
Ironsides"  Soldier  familiarizes 
himself  with  the  M249  Squad 
Automatic  Weapon  user  interface 
for  Virtual  Battle  Space  3  during 
an  Early  Synthetic  Prototyping 
pilot  test  held  on  Fort  Bliss, 
Texas.  (Photo  by  Sgt.  Brooks 
Fletcher,  16th  Mobile  Public 
Affairs  Detachment/Released) 
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Innovation  /  Training  /  Inception 


How  We’ll  Design  and  Manufacture  Systems  in  2025 


Pre-Engineered  Plug  and 
Play  Vehicle  Templates 


Persistent  Synthetic 
Gaming  Environments 
(Soldier  Crowd  Sourcing) 


Layered  Manufacturing,  Repai  , 
and  Logistics 

(FOB  Instant,  Regional  Rapid) 


Detailed  Engineering 


Customized  Mission-Optimal 
Ground  System 


3D  Virtual  World 
Acquired  on  Demand 


Manufacture/ 

Deployment 


*■  ■  ,-f- 


V  % 


Collaborative  3-d 
Immersive  Design 
Environment 


Semi-Autonomous 
Virtual  Prototype 
Engineering 

(proactive  M&S  that  does 
design  and  optimization) 


Physical  M&S  and 
Prototypes  (Full  Physics) 


Smith,  Robert  E.,  and  Brian  D.  Vogt.  A  Proposed  2025  Ground  Systems,  Systems  Engineering  Process.  DEFENSE  ACQUISITION 
UNIV  FT  BELVOIR  VA,  2014.  http://www.dtic. mil/cgi-bin/GetTRDoc?Location=U2&doc=GetTRDoc.pdf&AD-ADA608885 
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IMPS  Pilot  Study:  Robotic  Wingman 


•  Robotic  wingman  based  on  actual  demonstrator  system 

•  Three  scenarios: 

1 .  Track  a  red  convoy  (Al)  to  a  specific  location,  then  eliminate  it.  4  blue 

2.  Assault  a  defended,  fixed  location  to  free  prisoners.  2  blue/  2  red 

3.  Defend  an  urban  location  for  five  minutes.  2  blue/  2  red 


Big  Takeaways: 

•  Soldiers  very  enthusiastic  about  playing  game  -  especially  head-to-head 

•  Game  interface  is  very  important  (which  key  does  what) 

•  Scenarios  showed  definite  desire  to  tailor  platform  for  mission 


Source: 

http://calhoun.nps.edu/bitstream/handle/10945/44627/14Dec  Murray  Kate.pdf 
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Commercial  Gaming  Example  -  Call  of  Duty 


Per  Activision  (2013): 


*  100  million  players  (larger  than  the  population  of  the  UK,  Germany,  or  France) 


*  2.85  million  years  have  cumulatively  been  played 
(longer  than  the  entirety  of  human  existence.) 

•  NOT  PHYSICS  BASED  /  REALISTIC! ! ! 


ttttt  r 
ttttt  1  2 


THE  TOTAL  NUMBER  OF  PEOPLE  THAT 
HAVE  PLAYED  CALL  OF  DUTY*  IS  LARGER 
THAN  SOME  OF  THE  MOST  POPULATED 
COUNTRIES  IN  THE  WORLD 


2013  Cflll'DUTY 

W  ^  BY  THE  NUMBERS 


2  5  BILLION  HOURS  PLAYED 


,  I  t  .  1  □□  MILLION  f 

*  'ft1  »  -  * 


GERMANY  FRANCE  UNITED  KINGDOM  ITALY 

8D.4  MILLION  65.7  MILLION  63.7  MILLION  59.7  MILUON 


32T3aa,DQDTDOD, 


TH  AT*  S  32.3  QUADRILLION... 


|  -2.85  MILLIDN  VEAR5 

THE  TOTAL  HOURS  SPENT  PLAYING  CALL  OP  DUTY® 

IS  LONGER  THAN  THE  ENTIRETY  DF 

HUMAN  EXISTENCE 


http://games.mxdwn.com/news/over-100-million-people-have-played-call-of-duty/ 
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Relative  Feature  Priority  /  Value  Functions 


Priority  weightings  and  value 
functions  for  each  objective  are 
well  reasoned  based  on  SME  input 
and  gaming  data. 


Relative  Priority  Swing  Weight 

Weighting  Matrix 

Indicator  (Performance  Gap 

vs.  Importance) 


Value 

Function 

(knee  in  the 
curve) 


Supporting 

Narrative 


Source:  WSTAT  (Whole  Systems  Trades  Analysis  Tool)  Web:http://www.sandia.gov/CSR/assets/documents/WSTAT.pdf 
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TARDEC  Man  or  Hardware  in-The-Loop  Facilities 


US  ArmyTARDEC's  Ride  Motion 
Simulator  (RMS)  is  an  example  of  a  man- 
in-the-loop  physical  simulation. 


US  Army  TARDEC's  N-post  shaker  is  a 
hardware-in-the-loop  simulation. 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 


Further  References  Related  to  Tactical  Behavior  Mining  / 
Spatio-Temporal  Pattern  Recognition 


Ayers,  Jeanine,  and  Nick  Caler.  Automated  Support  for  After  Action  Review  (AAR)  Presentation.  No.  AP-R-1456.  APTIMA  INC  WOBURN  MA,  2009.  www.dtic.mil/ccii- 
bin/GetTRDoc?AD=ADA507982 

Bialkowski,  Alina,  et  al.  "Identifying  team  style  in  soccer  using  formations  learned  from  spatiotemporal  tracking  data."  Data  Mining  Workshop  (ICDMW),  2014  IEEE 
International  Conference  on.  IEEE,  2014.  https://s3-us-west-1.amazonaws.com/disnevresearch/wp-content/uploads/20150122Q55122/ldentifvinq-Team-Stvle-in-Soccer- 
usinq-Formations-from-Spatiotemporal-Trackinq-Data-Paper.pdf 

Bialkowski,  Alina,  et  al.  "Representing  Team  Behaviours  from  Noisy  Data  Using  Player  Role."  Computer  Vision  in  Sports.  Springer  International  Publishing,  2014.  247-269. 
https://s3-us-west-1.amazonaws.com/disnevresearch/wp-content/uploads/20150307075804/Representinq-Team-Behaviours-from-Noisv-Data-usinq-Plaver-Role-Paper.pdf 

Bose,  Guillaume,  et  al.  Strategic  pattern  discovery  in  rts-games  fore-sport  with  sequential  pattern  mining.  Tech.  Rep.  RR-LIRIS-2013-012,  2013. 
http://liris.cnrs.fr/Documents/Liris-6365.pdf 

Bowman,  Brian,  Niklas  Elmqvist,  and  T.  J.  Jankun-Kelly.  "Toward  visualization  for  games:  Theory,  design  space,  and  patterns."  Visualization  and  Computer  Graphics,  IEEE 
Transactions  on  18.11  (2012):  1956-1968.  http://www.umiacs.umd.edu/~elm/proiects/visqames/visqames.pdf 

Canossa,  A.,  M.  S.  El-Nasr,  and  THD  NGUYEN.  "Beyond  Visualization:  Democratizing  Access  to  Game  Analytics  Through  Interactive  Sense-making."  CHI  PLAY  Game  User 
Research  Workshop.  2014. 

Centieiro,  Pedro,  Teresa  Romao,  and  A.  Eduardo  Dias.  "Enhancing  Remote  Spectators’  Experience  During  Live  Sports  Broadcasts  with  Second  Screen  Applications."  More 
Playful  User  Interfaces.  Springer  Singapore,  2015.  231-261. 

Cheong,  Yun-Gyung,  et  al.  "Automatically  Generating  Summary  Visualizations  from  Game  Logs."  AIIDE.  2008.  https://www.aaai .orq/Papers/AI I DE/2008/AI I DE08-028. pdf 
Cottrell,  David.  "System  and  method  for  automated  creation  of  video  game  highlights."  U.S.  Patent  No.  8,515,253.  20  Aug.  2013. 
https://www.qooqle.com/patents/US8515253 

Dominguez,  Mike,  R.  Michael  Young,  and  Stephen  Roller.  "Design  and  Evaluation  of  Afterthought,  A  System  that  Automatically  Creates  Highlight  Cinematics  for  3D  Games." 
AIIDE.  201 1 .  https://www.aaai .orq/ocs/index. php/AI  I DE/AI I DE 1 1  /paper/download/4059/4425 

Drachen,  Anders,  et  al.  "A  comparison  of  methods  for  player  clustering  via  behavioral  telemetry."  arXiv preprint  arXiv.l 407.3950  (2014).  http://arxiv.org/pdf/1407.3950.pdf 
El-Nasr,  Magy  Seif,  Anders  Drachen,  and  Alessandro  Canossa.  Game  analytics:  Maximizing  the  value  of  player  data.  Springer  Science  &  Business  Media,  2013. 

El-Nasr,  Magy  Seif,  et  al.  "Data-Driven  Retrospective  Interviewing  (DDRI):  A  proposed  methodology  for  formative  evaluation  of  pervasive  games."  Entertainment  Computing 
11  (2015):  1-19. 

Gonzalez,  Avelino  J.,  and  Patrick  Brezillon.  "Comparing  two  context-driven  approaches  for  representation  of  human  tactical  behavior."  The  Knowledge  Engineering  Review 
23.03  (2008):  295-315.  http://www-svsdef.lip6.fr/~brezil/Paqes2/Publications/KER-2008-CxG-CxBR.pdf 

Gonzalez,  Avelino  J.,  and  Robert  Ahlers.  "Context-based  representation  of  intelligent  behavior  in  training  simulations."  Transactions  of  the  Society  for  Computer  Simulation 
15.4  (1998):  153-166.  https://www.researchqate.net/profile/Avelino  Gonzalez/publication/234791 270  Context- 
based  representation  of  intelligent  behavior  in  training  simulations/links/55563f5208ae6fd2d82361 10.pdf 

Gonzalez,  Avelino  J.,  Jose  Castro,  and  William  Gerber.  "Automating  the  acquisition  of  tactical  knowledge  for  military  missions."  The  Journal  of  Defense  Modeling  and 

Simulation:  Applications,  Methodology,  Technology  2.3  (2005):  145-160.  http://www.scs.org/pubs/idms/vol2num3/Gonzalez-Castro-pp145-160.pdf 

Good,  Owen.  "Dota  2  is  Steam's  first  game  with  1  million  users  playing  at  the  same  time".  Polygon.  http://www. polygon . com/20 1 5/2/1 5/8042 1 7 1  /dota-2-concu rrent-users- 

million-steam-pc 

Ha,  Eunyoung,  et  al.  "Goal  Recognition  with  Markov  Logic  Networks  for  Player-Adaptive  Games."  AIIDE.  201 1 . 
http://www.aaai  .orq/ocs/index.  php/AI  I  DE/AI  I  DE  1 1  /paper/download/4077/4409/ 

Hsueh,  Pei-Yun,  Prem  Melville,  and  Vikas  Sindhwani.  "Data  quality  from  crowdsourcing:  a  study  of  annotation  selection  criteria."  Proceedings  of  the  NAACL  HLT  2009 
workshop  on  active  learning  for  natural  language  processing.  Association  for  Computational  Linguistics,  2009.  http://www.aclweb.org/website/old  antholoqy/W/W09/W09- 
19.pdf#paqe=37 

Jensen,  Randy,  et  al.  Visually  based  timeline  debrief  toolset  for  team  training  AAR.  STOTTLER  HENKE  ASSOCIATES  INC  SAN  MATEO  CA,  2006. 
https://www.stottlerhenke.com/papers/IITSEC-06-visual-timeline-debrief-team-traininq-aar.pdf 

Johnson,  Cynthia,  and  Avelino  J.  Gonzalez.  "Automated  after  action  review:  State-of-the-art  review  and  trends."  The  Journal  of  Defense  Modeling  and  Simulation:  Applications,  Methodology, 
Technology  5.2  (2008):  108-121.  http://www. scs. ora/Dubs/idms/vol5num2/Johnson-AAR-final. pdf 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 


35 


Further  References  Related  to  Tactical  Behavior  Mining  / 
Spatio-Temporal  Pattern  Recognition  Continued 


Kooij,  Julian  FP,  Gwenn  Englebienne,  and  Dariu  M.  Gavrila.  "A  non-parametric  hierarchical  model  to  discover  behavior  dynamics  from  tracks."  Computer  Vision-ECCV  2012. 
Springer  Berlin  Heidelberg,  2012.  270-283.  http://www.lookinqatpeople.com/eccv12.pdf 

Korfiatis,  Peter,  and  Robert  Cloutier.  "Using  3D  gaming  technologies  to  improve  the  Concept  of  Operations  (CONOPS)  process."  Presentation  to  the  National  Defense 
Industrial  Association's  Ground  Vehicle  Systems  Engineering  and  Technology  Symposium,  Troy,  Ml,  August.  2013.  http://serc2cdn1.mannadesiqnworks.netdna- 
cdn.com/wp-content/uploads/201 4/03/94  Usinq-3D-Gaminq  NDIA  Final-paper.pdf 

Lasecki,  Walters.,  et  al.  "Real-time  crowd  labeling  for  deployable  activity  recognition."  Proceedings  of  the  2013  conference  on  Computer  supported  cooperative  work.  ACM, 
2013.  http://ftp.cs.rochester.edU/u/vsonq/papers/cscw-ar.pdf 

Leban,  Gregor,  et  al.  "Vizrank:  Data  visualization  guided  by  machine  learning."  Data  Mining  and  Knowledge  Discovery  13.2  (2006):  119-136. 

Leece,  Michael  A.,  and  Arnav  Jhala.  "Sequential  pattern  mining  in  starcraft:  brood  war  for  short  and  long-term  goals."  Tenth  Artificial  Intelligence  and  Interactive  Digital 
Entertainment  Conference.  2014.  http://www.aaai .orq/ocs/index. php/AI I DE/AI I DE 1 4/paper/viewFile/9082/90 1 1 

Lucey,  Patrick,  et  al.  "Assessing  team  strategy  using  spatiotemporal  data."  Proceedings  of  the  19th  ACM  SIGKDD  international  conference  on  Knowledge  discovery  and 
data  mining.  ACM,  2013.  http://www.disnevresearch.com/wp-content/uploads/PROJECT  AssessinqTeamStrateqy  kdd2013  paper.pdf 

Lucey,  Patrick,  et  al.  "Representing  and  discovering  adversarial  team  behaviors  using  player  roles."  Proceedings  of  the  IEEE  Conference  on  Computer  Vision  and  Pattern 
Recognition.  2013.  https://s3-us-west-1  .amazonaws.com/disnevresearch/wp-content/uploads/20141 125022728/Representinq-and-Discoverinq-Adversarial-Team-Behaviors- 
Usinq-Plaver-Roles-Paper.pdf 

Mahlmann,  Tobias,  Matthias  Schubert,  and  Anders  Drachen.  "Esports  Analytics  Through  Encounter  Detection."  MIT  Sloan  Sports  Analytics  Conference.  MIT  Press,  2016. 
www.sloansportsconference.com/wp-content/uploads/2016/02/1458.pdf 

Miyamori,  Hisashi,  and  Shun-ichi  lisaku.  "Video  annotation  for  content-based  retrieval  using  human  behavior  analysis  and  domain  knowledge."  Automatic  Face  and  Gesture 
Recognition,  2000.  Proceedings.  Fourth  IEEE  International  Conference  on.  IEEE,  2000.  ftp://ftp.cfar.umd.edU/.snapshot/niqhtlv.0/pub/aravinds/00840653.pdf 
Nguyen,  Truong-Huy  Dinh,  Magy  Seif  El-Nasr,  and  Alessandro  Canossa.  "Glyph:  Visualization  Tool  for  Understanding  Problem  Solving  Strategies  in  Puzzle  Games." 
Foundations  of  Digital  Games  (FDG)  (2015). 

Sha,  Long,  et  al.  "Chalkboarding:  A  New  Spatiotemporal  Query  Paradigm  for  Sports  Play  Retrieval."  Proceedings  of  the  21st  International  Conference  on  Intelligent  User 
Interfaces.  ACM,  2016.  https://www.disnevresearch.com/publication/chalkboardinq/ 

Stensrud,  Brian  S.,  Gilbert  C.  Barrett,  and  Avelino  J.  Gonzalez.  "Context-Based  Reasoning:  A  Revised  Specification."  FLAIRS  Conference.  2004. 
http://www.aaai.org/Papers/FLAIRS/2004/Flairs04-104.pdf 

Sun,  Guo-Dao,  et  al.  "A  survey  of  visual  analytics  techniques  and  applications:  State-of-the-art  research  and  future  challenges."  Journal  of  Computer  Science  and 
Technology  28.5  (2013):  852-867.  www.cad.ziu.edu.cn/home/vcwu/Files/va  survev.pdf 

Synnaeve,  Gabriel,  and  Pierre  Bessiere.  "A  Bayesian  model  for  plan  recognition  in  RTS  games  applied  to  StarCraft."  a rXiv preprint  arXiv:1 11 1.3735  (2011). 
http://arxiv.Org/pdf/1 1 1 1 .3735 

Tadjdeh,  Yasmin,  “New  Video  Game  Could  Speed  Up  Acquisition  Timelines.”  National  Defense  Magazine  blog  post  dated  12/3/2014: 
http://www.nationaldefensemaqazine.orq/bloq/lists/posts/post.aspx?ID=1687 

Tastan,  Bulent,  and  Gita  Reese  Sukthankar.  "Learning  Policies  for  First  Person  Shooter  Games  Using  Inverse  Reinforcement  Learning."  AIIDE.  2011. 
http://citeseerx.ist.psu.edU/viewdoc/download7doM0.1 .1.384.5902&rep=rep1&type=pdf 

TrueSight  Visual  Tool  for  Analyzing  Video  Game  Play  .  http://tunqlinwu.com/assets/portfolio/.../lnfoVis  Mid  Phase  Project  Design  Report.pdf 

van  Lent,  Michael,  and  John  E.  Laird.  "Learning  procedural  knowledge  through  observation."  Proceedings  of  the  1st  international  conference  on  Knowledge  capture.  ACM, 
2001.  https://www.researchqate.net/profile/John  Laird/publication/2925010  Learning  Procedural  Knowledge  through  Qbservation/links/5464efa80cf2052b509f28a1  .pdf 

Van  Lent,  Michael,  and  John  Laird.  "Learning  by  observation  in  complex  domains."  Ann  Arbor  1001  (1998):  48109. 
http://web.eecs.umich.edu/~soar/sitemaker/workshop/19/vanlent-SWS-Thesis.pdf 

Van  Lent,  Michael,  and  John  Laird.  "Learning  hierarchical  performance  knowledge  by  observation."  Ann  Arbor  1001  (1999):  48109-2110. 
http://ai.eecs.umich.edu/people/laird/papers/ML99-2.pdf 

Vondrick,  Carl,  Donald  Patterson,  and  Deva  Ramanan.  "Efficiently  scaling  up  crowdsourced  video  annotation."  International  Journal  of  Computer  Vision  101.1  (2013):  184- 
204.  http://cvrr.ucsd.edu/ece285/Sprinq2014/papers/Vondrick  IJCV2013.pdf 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release;  distribution  is  unlimited. 


36 


